Inline handwriting recognition and correction

ABSTRACT

As a user writes using a handheld writing device, such as an electronic pen or stylus, handwriting input is received and initially displayed as digital ink. The display of the digital ink is converted to recognized text inline with additional digital ink as the user continues to write. A user may edit a word of recognized text inline with other text by selecting the word. An enlarged version of the word is displayed in a character correction user interface that allows a user to make corrections on an individual character basis and also provides other correction options for the word.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/962,438, filed Dec. 21, 2007, which is herein incorporated byreference in its entirety.

BACKGROUND

Typical computer systems, especially computer systems using graphicaluser interface (GUI) systems, are optimized for accepting user inputfrom one or more discrete input devices. Common input devices include akeyboard for entering text and a pointing device, such as a mouse withone or more buttons, for controlling the user interface. The keyboardand mouse interface facilitates user interaction with computer systemssuch as for the creation and modification of electronic documentsincluding text, spreadsheets, database fields, drawings, and photos.

One of the limitations with conventional GUI systems is that a user mustgenerally type the text they are entering into the computer system usingthe keyboard. More recently, however, pen-based computing systems, suchas tablet PCs and the like, have been increasing in popularity. Inpen-based computing systems, user input advantageously may be introducedusing an electronic “pen” or stylus. More particularly, a user can writewith the stylus on the screen of the electronic tablet or otherappropriate surface in a similar manner to traditional pen and paper.The strokes of the user's handwriting with the stylus are read and usedto recreate the handwriting in electronic form on the screen with“digital ink.”

As pen-based computing systems are becoming more popular, users areincreasingly entering more data in the form of digital ink. In manyinstances, users may wish to convert the original digital ink data tomachine-generated text, i.e., text suitable for use and manipulation byconventional word processing programs and other application programs.While handwriting recognition technology for converting handwrittendigital ink to machine-generated text has improved in recent years,recognizers of this type still are error prone, particularly for userswith poor handwriting, users that write at an angle, and/or users thatwrite very quickly. As a result, machine-generated text produced by arecognizer, at times, needs to be corrected and/or otherwise changed oredited by the user.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Embodiments of the present invention relate to providing inlinehandwriting recognition and correction. When a user writes using ahandheld writing device, such as an electronic pen or stylus, thehandwriting input is initially displayed as digital ink. The handwritinginput is also analyzed using a recognizer to identify one or more wordsof text for the digital ink. As the user continues to write, the displayof the handwriting input may be converted to the text from therecognizer. The text replaces the digital ink such that the text isinline with further digital ink provided as the user continues to write.Additionally, a user may correct a word of text converted from digitalink by selecting the word. Based on the selection, the word is presentedin a correction user interface inline with the other text. Thecorrection user interface provides an enlarged view of the word in placeof an original view of the word and allows editing of the word on anindividual character basis, as well as providing additional correctionoptions for the word. When the word is displayed in the correction userinterface, the display of the other words of text may be adjusted toprovide space for displaying the correction user interface inline withthe other words.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in detail below with reference to theattached drawing figures, wherein:

FIG. 1 is a block diagram of an exemplary computing environment suitablefor use in implementing the present invention;

FIGS. 2A-2C are screen displays illustrating conversion of digital inkto recognized text in accordance with an embodiment of the presentinvention;

FIG. 3 is a screen displaying illustrating distance triggers for wordrecognition in accordance with an embodiment of the present invention;

FIGS. 4A-4C are screen displays illustrating the use of saved digitalink to correct text recognition in accordance with an embodiment of thepresent invention;

FIGS. 5A and 5B are screen displays illustrating deletion of a spacewhile in word-mode in accordance with an embodiment of the presentinvention;

FIGS. 6A and 6B are graphs showing word spacing size and word spacingshrink speed, respectively, versus time in accordance with an embodimentof the present invention;

FIGS. 7A-7C are screen displays illustrating the addition of handwritingbetween recognition text in accordance with an embodiment of the presentinvention;

FIG. 8 is a screen display illustrating a user hovering a pen over aword of text in accordance with an embodiment of the present invention;

FIG. 9 is a screen display illustrating drop-down arrow provided undereach word box to indicate that each word may be selected in accordancewith an embodiment of the present invention;

FIG. 10 is a screen display illustrating a word in character-mode inaccordance with an embodiment of the present invention;

FIG. 11 is a screen display illustrating a user writing a handwrittencharacter over a character of recognition text in character-mode inaccordance with an embodiment of the present invention;

FIG. 12 is a screen display illustrating a menu to provide furtherfeatures in character-mode in accordance with an embodiment of thepresent invention;

FIG. 13 is a screen display illustrating an expander button providedwith characters to allow the user to access further single-characterfeatures for each character in accordance with an embodiment of thepresent invention;

FIG. 14 is a screen display illustrating single-character features for acharacter selected in character-mode in accordance with an embodiment ofthe present invention;

FIGS. 15A-15D are screen displays illustrating an automatic wordcorrection based on a user's editing in character-mode in accordancewith an embodiment of the present invention;

FIGS. 16A-16H are screen displays illustrating adjusting font size, wordpositions, word spacing, and use of scrolls to provide space for a wordin character-mode in accordance with an embodiment of the presentinvention;

FIG. 17 is a flow diagram showing a method for automatically convertingdigital ink to recognized text in accordance with an embodiment of thepresent invention; and

FIG. 18 is a flow diagram showing a method for displaying a selectedword in a correction user interface in accordance with an embodiment ofthe present invention.

DETAILED DESCRIPTION

The subject matter of the present invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

Overview

As noted previously, although handwriting recognition technology hasimproved over recent years, handwriting recognition is not perfectlyaccurate and may be prone to errors in recognition for a variety ofreasons, such as poor handwriting, users writing at an angle, and/orusers writing very quickly. Accordingly, users will continue to need tobe able to make corrections to text recognized from the users'handwriting. Embodiments of the present invention provide an efficientinline handwriting recognition and correction user experience. Theapproach speeds up the text input and correction process and makes itfeel less tedious for the user.

In accordance with embodiments of the present invention, as a userwrites using a handheld writing device such as an electronic pen orstylus, digital ink is initially displayed, and the digital ink isconverted to text directly within the writing area as the user continuesto write. In some embodiments, the system attempts to convert digitalink to text each time a user finishes writing a word and/or beginswriting a new word. This provides the user with a visual indication ofthe conversion that is occurring and also makes it possible for the userto append more digital ink at the correct location even if the ink isconverted to text too early (e.g., in the middle of a word). Althoughdigital ink is no longer displayed after it has been converted to text,in some embodiments, the digital ink may be saved in the background andused for recognition purposes as the user continues to write.

In some embodiments, the user may also add handwriting between or beforewords of recognized text. In particular, the user may begin to write inthe space between two words of recognized text or before a first word ofrecognized text, and the space continues to grow as the user writes toprovide room for the user's handwriting. As the user continues to write,the digital ink is converted to text. When the user finishes writing,the space shrinks back down to a default space size between words.

Embodiments of the invention also provide for inline correction, whichenables a user to correct handwriting recognition errors directly in thewriting area. When a user selects a word of recognized text, the word ispresented using a correction user interface. In particular, an enlargedview of the word is provided by “zooming in” on the word. The correctionuser interface allows the user to make corrections to the word on anindividual character basis and may also present additional options forcorrecting the word. To provide space for the enlarged view of aselected word inline with other words of text, the display of the otherwords may be adjusted, for instance, by resizing and/or moving the otherwords.

Accordingly, in one aspect, an embodiment of the present invention isdirected to one or more computer-storage media embodyingcomputer-useable instructions for performing a method. The methodincludes receiving input corresponding with user handwriting from ahandheld writing device and displaying digital ink representing the userhandwriting based on the input. The method also includes analyzing theinput using a recognizer to identify one or more words as recognitiontext for the digital ink. The method further includes employing at leastthree triggers to determine when to convert display of the digital inkto display of the recognition text. The three triggers include adistance-based trigger, a recognition-based trigger, and an overalltimer-based trigger. The method still further includes determining thatat least one of the triggers has been satisfied indicating to convertdisplay of the digital ink to the recognition text, and displaying therecognition text in place of the digital ink.

In another aspect, an embodiment of the present invention relates to oneor more computer-storage media embodying computer-useable instructionsfor performing a method. The method includes displaying text includingtwo or more words, wherein the text was previously converted fromdigital ink corresponding with handwriting input. The method alsoincludes receiving a user selection of a word within the text. Themethod further includes responsive to the user selection, displaying theword in a correction user interface allowing correction of the word onan individual character basis. The correction user interface provides anenlarged display of the word in place of an original display of the wordin a manner in which the enlarged display of the word is inline withother words within the text. The method still further includes adjustingthe display of the other words within the text to provide space fordisplaying the correction user interface including the enlarged displayof the word.

In a further aspect of the invention, an embodiment is directed to oneor more computer-storage media embodying computer-useable instructionsfor performing a method. The method includes receiving handwriting inputcorresponding with a handheld writing device being moved relative to adisplay surface and displaying digital ink at a location on a displaycorresponding with a location of the display surface at which thehandwriting input is received. The method also includes analyzing thehandwriting input using a recognizer to identify one or more words oftext corresponding with the handwriting input. The method furtherincludes replacing display of the digital ink with the text as the usercontinues to provide additional handwriting input using the handheldwriting device, wherein the text is displayed inline with additionaldigital ink corresponding with the additional handwriting input. Themethod also includes receiving input to present a word of the text in acharacter correction interface. The character correction interfaceprovides an enlarged display of the word in place of an original displayof the word to facilitate receiving further handwriting input to correctthe word. The character correction interface also allows correction ofthe word on an individual character basis. The method further includesdisplaying the word in the character correction interface by replacingthe original display of the word with the enlarged display of the word.The method still further includes adjusting display of other wordswithin the text to provide space for the enlarged display of the word.

Exemplary Operating Environment

Having briefly described an overview of the present invention, anexemplary operating environment in which various aspects of the presentinvention may be implemented is described below in order to provide ageneral context for various aspects of the present invention. Referringinitially to FIG. 1 in particular, an exemplary operating environmentfor implementing embodiments of the present invention is shown anddesignated generally as computing device 100. Computing device 100 isbut one example of a suitable computing environment and is not intendedto suggest any limitation as to the scope of use or functionality of theinvention. Neither should the computing device 100 be interpreted ashaving any dependency or requirement relating to any one or combinationof components illustrated.

The invention may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program modules, being executed by a computer orother machine, such as a personal data assistant or other handheldwriting device. Generally, program modules including routines, programs,objects, components, data structures, etc., refer to code that performparticular tasks or implement particular abstract data types. Theinvention may be practiced in a variety of system configurations,including hand-held devices, consumer electronics, general-purposecomputers, more specialty computing devices, etc. The invention may alsobe practiced in distributed computing environments where tasks areperformed by remote-processing devices that are linked through acommunications network.

With reference to FIG. 1, computing device 100 includes a bus 110 thatdirectly or indirectly couples the following devices: memory 112, one ormore processors 114, one or more presentation components 116,input/output ports 118, input/output components 120, and an illustrativepower supply 122. Bus 110 represents what may be one or more busses(such as an address bus, data bus, or combination thereof). Although thevarious blocks of FIG. 1 are shown with lines for the sake of clarity,in reality, delineating various components is not so clear, andmetaphorically, the lines would more accurately be grey and fuzzy. Forexample, one may consider a presentation component such as a displaydevice to be an I/O component. Also, processors have memory. Werecognize that such is the nature of the art, and reiterate that thediagram of FIG. 1 is merely illustrative of an exemplary computingdevice that can be used in connection with one or more embodiments ofthe present invention. Distinction is not made between such categoriesas “workstation,” “server,” “laptop,” “hand-held device,” etc., as allare contemplated within the scope of FIG. 1 and reference to “computingdevice.”

Computing device 100 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 100 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprise computerstorage media and communication media. Computer storage media includesboth volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computing device 100. Communication mediatypically embodies computer-readable instructions, data structures,program modules or other data in a modulated data signal such as acarrier wave or other transport mechanism and includes any informationdelivery media. The term “modulated data signal” means a signal that hasone or more of its characteristics set or changed in such a manner as toencode information in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of any of the aboveshould also be included within the scope of computer-readable media.

Memory 112 includes computer-storage media in the form of volatileand/or nonvolatile memory. The memory may be removable, nonremovable, ora combination thereof. Exemplary hardware devices include solid-statememory, hard drives, optical-disc drives, etc. Computing device 100includes one or more processors that read data from various entitiessuch as memory 112 or I/O components 120. Presentation component(s) 116present data indications to a user or other device. Exemplarypresentation components include a display device, speaker, printingcomponent, vibrating component, etc.

I/O ports 118 allow computing device 100 to be logically coupled toother devices including I/O components 120, some of which may be builtin. Illustrative components include a microphone, joystick, game pad,satellite dish, scanner, printer, wireless device, keyboard, or a mouse.A pen digitizer and an accompanying pen or stylus may be provided inorder to digitally capture freehand input (e.g. electronic ink). The pendigitizer may be connected to a serial port interface, directly to theprocessor 114, or it may be coupled to the processor 114 in any suitablemanner, such as via a parallel port or another interface and the systembus 110 as is known in the art. Furthermore, the usable input area ofthe digitizer may be integrated in a monitor or similar presentationcomponent 116, or it may exist as a separate device overlaying orotherwise appended to a monitor or similar presentation component 116.Using a handheld writing device, such as a stylus, a user can select,highlight, and write on a digitizing display area. Examples of suitabledigitizing display panels include electromagnetic pen digitizers andoptical digitizers. Other types of pen digitizers may also be used. Thedigitizer, in connection with the computing system interprets gesturesmade using the stylus in order to manipulate data, enter text, andexecute conventional computer application tasks, such as creating,editing, and modifying spreadsheets, word processing programs, and thelike.

The stylus may be equipped with buttons or other features to augment itscapabilities. In one example, a stylus could be implemented as a“pencil” or “pen”, in which one end constitutes a writing portion andthe other end constitutes an “eraser” end, and which, when moved acrossthe display, indicates portions of electronic ink on the display thatare to be erased. Other types of input devices, such as a mouse,trackball, keyboard, or the like also could be used as a handheldwriting device in the system of FIG. 1. Additionally, a user's ownfinger could be used for selecting or indicating portions of thedisplayed image on a touch-sensitive or proximity-sensitive display.Consequently, the term “user input device,” as used herein, is intendedto have a broad definition and encompasses many variations on well-knowninput devices.

In various examples, the system provides an ink platform as a set of COM(component object model) services that an application program can use tocapture, manipulate, and store ink. The ink platform also may include amark-up language including a language like the extensible markuplanguage (XML). As still further examples, systems may use “.NET” orDCOM (distributed component object model) as additional implementationsof the ink platform. Yet further implementations may be used includingthe Win32 programming model and the .Net programming model formMicrosoft Corporation. These platforms are commercially available andknown in the art.

In addition to use with full performance pen-based computing systems or“tablet PCs” (e.g. convertible laptops or “slate” type tablet PCs),aspects of this invention can be used in conjunction with other types ofpen-based computing systems and/or other devices that accept and/ordisplay data as digital ink and/or accept and/or display electronic penor stylus input, such as: hand-held or palm-top computing systems;personal digital assistants; pocket personal computers; mobile andcellular telephones, pagers, and other communications devices; watches;appliances; and any other devices or systems that include a monitor orother display device and/or digitizer that presents printed or graphicalinformation to users and/or allows input using an electronic pen orstylus.

The computing device 100 may also include or be coupled to a digital inkprocessing software module. The ink processing module is operable forreceiving data from the digitizer and/or the stylus and rendering thatdata as digital ink (or “wet ink”). Wet digital ink comprises ink thathas not been sent to a recognizer to be converted to text. In contrast,“dry ink” comprises digital ink that has been converted to text by arecognizer.

In one exemplary embodiment, the ink processing module comprises acollection of software modules that perform different tasks forrecognizing handwriting strokes as digital ink. For example, a stylusand ink module may be provided to receive data describing the positionsand angles of the stylus for a series of handwriting strokes. The stylusand ink module interprets the data for recognizing digital ink. Othersoftware modules, such as a recognizer identify certain handwritingstrokes and assign them a particular significance. For example, therecognizer converts digital ink to text. In an embodiment, therecognizer breaks the user's handwriting in digital ink down intoseparate word blocks, evaluates the word blocks, and then uponrecognizing a word contained in each word block, converts the digital toword-based text (i.e. text that is typically viewed on a computermonitor or other display device). Additionally, certain gestures (suchas a cross-out) may be recognized and associated with other editingprocesses. The ink processing module can also include an erasingfunctions module for removing digital ink that has been previouslyrendered. Those skilled in the art will recognize that the computingdevice 100 may include or be coupled to other software modules fordigital ink processing.

Inline Handwriting Recognition and Correction

As will be described in further detail below, operation of embodimentsof the present invention may be considered to occur within two modes:word-mode and character-mode. Word-mode is mainly used to write newwords. A user may write words in a variety of different locations,including, for instance, after the rightmost text, between text words,or before the first text word. As the user writes, digital ink isrecognized and the display of the digital ink is converted to text fromthe recognizer. From word-mode, the user can drill down tocharacter-mode to display a correction user interface for a single word.Character-mode allows the user to make corrections to the word inlinewith other words of recognized text.

Word-Mode

In word-mode, digital ink is recognized using a recognizer as the useris writing, and the digital ink is converted to text directly in thewriting area. In an embodiment, the conversion from displayed digitalink to recognized text is performed in a manner such that a word ofdigital ink is converted to recognized text as the user finishes writingthe word and/or starts writing the next word. FIGS. 2A-2C providescreenshots illustrating the conversion of digital ink to recognizedtext in accordance with an embodiment of the present invention.

Referring initially to FIG. 2A, a user is writing in a writing area 202.A portion of the writing (“This is the new”) has already been convertedto recognized text, and the user is currently writing the word “writing”within the writing area 202. As shown in FIG. 2A, in an embodiment, aline 204 is provided within the writing area 202 indicating to the userwhere the user can write, and the line is removed as digital ink isconverted to recognized text. As the user is writing, digital ink 206(i.e., “writing”) is displayed within the writing area. The digital ink206 may be collected and stored in any suitable manner and using anysuitable format known in the art. The digital ink 206 is also sent to arecognizer, which attempts to recognize text that corresponds with thedigital ink 206.

Based on a number of triggers within various embodiments of theinvention as will be discussed in further detail below, as the user iswriting, a determination is made to convert the displayed digital ink206 within the writing area 202 to recognized text. As shown in FIG. 2B,in an embodiment, the conversion begins by displaying the recognizedtext (“writing”) 208 within the writing area 202. In an embodiment, thedigital ink 206 is initially displayed with the recognized text 208 butbegins to gradually fade away and disappears after a configurable timeinterval (e.g., after 1000 ms). FIG. 2C illustrates the writing area 202after the conversion from digital ink 206 to recognized text 208 hascompleted. As shown in FIG. 2C, only the recognized text 208 ispresented within the writing area, and the digital ink 206 has beenremoved.

When digital ink is converted to recognized text, the text showing therecognized handwriting is divided into separate word boxes for eachword. For instance, with reference to FIG. 2C, a word box correspondswith each word in the recognized text, including: “This,” “is,” “the,”“new,” and “writing.” In an embodiment, when the digital ink isconverted into word boxes, the system avoids displaying a word box(e.g., the recognized text) under the user's current pen location as theuser is continuing to write such that the user is not distracted bywriting over text. For instance, with reference to FIG. 2B, if the wordbox for the recognized text “writing” 208 exceeded the digital ink 206corresponding with the word box, the user would be distracted by therecognized text as the user continues to write.

To avoid this to the extent possible, the size of word boxes may bedetermined such that the rightmost word box does not exceed therightmost digital ink stroke. By way of example only and not limitation,in one embodiment, this may be determined by first drawing the currentrecognized text with the currently used font size for the other wordboxes (or using a default font size, e.g., if this is the first wordbox). The width of the new text box for this text is measured andcompared against an available word box space. The available word boxspace is defined as the distance between the rightmost word box (or theleft edge of the writing surface if no word box is displayed yet) andthe rightmost ink stroke. If the available word box space is greaterthan or equal to the newly created word box width, then the new word boxis inserted without modification. On the other hand, if the new word boxis wider than the available word box space, the new word box and/orother word boxes may be scaled such that the new word box does notexceed the available word box space. For instance, in an embodiment, allthe word boxes to the left and the new word box are scaled down so thatthey will align with the rightmost ink stroke. If there are any wordboxes to the right (e.g., when the user is inserting a word between twowords of recognized text) then those word boxes may also be updated touse the reduced font size. This will keep the font size for all the textboxes on this line consistent, and thus more readable. In an embodiment,a minimum font size is defined to ensure that word boxes are not reducedto such a small size that they are not readable. In the edge case wherethis minimum font size is reached, the word boxes might appear under theuser's pen location. Additionally, in an embodiment, a default size maybe used to define the word space between each word box (e.g., 25% of theheight of the writing surface). This word space can also shrink with thesize reduction of the word boxes, but may be kept consistent for all theword spaces to ensure readability.

As noted previously, the display of digital ink is converted to textinline as the user continues to write. In some embodiments, the systemattempts to convert each digital ink word to recognized text after theuser completes writing the word and/or begins to write the next word. Anumber of triggers may be employed for determining when to convert thedisplay of digital ink to text within the scope of embodiments of theinvention. In one embodiment, a combination of three triggers isemployed for determining when to convert digital ink to text. The threetriggers include: a distance-based trigger; a recognition-based trigger;and an overall timer-based trigger. Conversion of digital ink torecognized text within a writing area occurs whenever any of thesetriggers has been satisfied.

The distance-based trigger converts digital ink to recognized text ifthe user moves the pen a certain distance from the left-most orright-most ink strokes. In an embodiment, the distance-based triggerincludes two distance triggers. The first distance trigger is ashort-distance trigger that includes a time-out to avoid mistakenconversions if the pen just passes the short-distance slightly. Thesecond distance trigger is a long-distance trigger that converts digitalink to recognized text when the long-distance is passed.

The two trigger distances are illustrated in FIG. 3 as two rectangleboundaries 302 and 304. The smaller boundary 302 corresponds with theshort-distance trigger. The width of the smaller boundary 302 iscalculated based on a “trigger distance” 306. The trigger distance 306is defined as a configurable distance to the right of the user'srightmost ink stroke (e.g., X number of pixels to the right of theuser's rightmost ink stroke). Accordingly, the width of the smallerboundary 306 starts one trigger distance to the left of the leftmost inkstroke and ends one trigger distance to the right of the rightmost inkstroke. The height of the smaller boundary 302 may vary, but in anembodiment, the height of the smaller boundary 302 corresponds with theheight of the writing surface.

In an embodiment, a default value may be used to define the triggerdistance. For instance, a default trigger distance of 25 pixels may beemployed. In another embodiment, the trigger distance automaticallyadapts to fit the user's writing style by determining an average wordspace (i.e., spacing between words) for the user and setting the triggerdistance to the user's average word space. In particular, the word spacedistances from the user's digital ink (e.g., every time “wet” ink isdiscarded as will be described in further detail below) will be measuredand sent to a trigger distance resizing algorithm. The algorithm willrecalculate the user's average word space size, and the new value willbe used to update the trigger distance. This automatic adjustmentfacilitates avoiding the trigger distance from being too aggressive(e.g., conversion to text is performed in the middle of a word) or toopassive (e.g., conversion to text is not performed after each word iswritten). In an embodiment, a minimum trigger distance may be employedsuch that the trigger distance may never be lower than the minimumregardless of the user's average word space. Additionally, inembodiments, each user may have an associated trigger distance. In afurther embodiment, the length of the time-out associated with theshort-distance trigger may be adapted to fit the user's handwritingstyle by analyzing the typical time between writing words for the user.

The larger boundary 304 corresponds with the long-distance trigger. Inan embodiment, the larger boundary 304 is a preconfigured sizeindependent of the smaller boundary 302. In another embodiment, the sizeof the larger boundary 304 is based on the size of the smaller boundary302. For instance, the larger boundary 304 may be a predetermined numberof pixels bigger than the smaller boundary 302.

When the user lifts the pen and passes the larger boundary 304, alldigital ink is converted to text. In an embodiment, if the pen onlypasses the smaller boundary 302, but stays within the larger boundary304, a timer is started. If the user's pen stays in this region untilthe timer has passed a predetermined “crossing time” (e.g., 500 ms), thecurrent digital ink on the writing surface is converted to text. If theuser's pen moves back into the smallest rectangle boundary again, thenthe crossing-timer is reset. This crossing time is intended to preventaccidental conversions. On a device that doesn't support hover, theconversion may be triggered when the user starts inking outside of thesmallest rectangle boundary.

The recognition-based conversion employs the recognition results from arecognizer in an attempt to determine whether the user has finishedwriting a word and to convert the display of digital ink to recognizedtext. The recognition-based conversion may be time-based. In particular,if a word doesn't change in the recognition results within a set timeperiod (e.g., 500 ms), then a word box gets created for that word andthe corresponding ink is converted to the recognized text.Recognition-based conversion may be useful, for instance, if a distancetrigger is too big. In particular, as a user is writing, the user'sspacing between words may not exceed the distance-based triggers.However, the recognition results may indicate that a user has finishedwriting a word triggering the conversion of digital ink to recognizedtext.

The third conversion mechanism used in an embodiment of the presentinvention is an overall timer. This timer starts when the user lifts thepen, and is reset every time inking resumes. A predetermined overalltime may be established for conversion purposes (e.g., 1500 ms). Whenthe timer reaches the predetermined overall time, all digital ink isconverted to recognized text.

In an embodiment, digital ink that is removed from display within thewriting surface when conversion occurs is saved in the background forrecognition purposes. This makes it possible to append ink to a wordthat is converting or has been converted to recognized text. The digitalink that is no longer displayed or being removed but is saved in thebackground (referred to herein as “wet ink”) remains where the userwrote the digital ink so that additional digital ink written by the useris received in relation to the original ink location. For instance, ifdigital ink is converted to recognized text too early (e.g., in themiddle of a word), as the user continues to write, the saved digital inkmay be used with the new digital ink the user writes for recognitionpurposes. This allows the recognized text that was originally displayedto be changed based on further recognition.

By way of example with reference to FIGS. 4A-4C, suppose that a userintends to write the word “together.” As shown, in FIG. 4A, the userbegins writing the word “together” in the writing surface 402. In thepresent example, after the user has written the “toget” portion of theword “together,” the recognizer initially recognizes the “to” and “get”portions of the digital ink as separate words and converts the digitalink to recognized text, as shown in FIG. 4B. However, the system savesthe digital ink in the background. As the user continues to write the“her” portion of “together,” as shown in FIG. 4B, the recognizer usesthe new digital ink (“her”) as well as the saved digital ink (“toget”)to determine that the user intended to write the word “together.”Accordingly, as shown in FIG. 4C, instead of displaying “to,” “get,” and“her” as separate words, the word “together” is displayed as a singleword.

It may not be desirable to indefinitely save wet ink in the backgroundand use the wet ink for recognition purposes as this may cause thedisplayed recognition results to frequently change as the user continuesto writes, which may be distracting to the user. Accordingly, in someembodiments of the present invention, the system continues to save wetink in the background and use the wet ink for recognition purposes onlyuntil a particular condition occurs. When such a condition occurs, thewet ink may be discarded and/or removed from being actively consideredfor recognition purposes. This may help avoid changing displayedrecognition text too often. In various embodiments of the invention, wetink is discarded and/or no longer used for recognition purposesaccording to one or more of the following conditions.

-   -   The user goes into character-mode (as will be described in        further detail below) for any of the recognized words.    -   The user starts to write between any of the recognized words.    -   The writing surface gets cleared and/or closed.    -   The user starts writing a predetermined distance to the left of        the right-most ink-stroke. This prevents garbled wet ink that        would result from the saved ink being used in conjunction with        the new wet ink that is written on top of the saved wet ink.    -   The recognition results have been stabilized for a predetermined        period of time.    -   The wet ink is more than a predetermined number of words back.        In other words, a buffer zone comprising a predetermined number        of words may be employed. In an embodiment a buffer zone of        three words is employed. Any words outside of the buffer zone        (e.g., more than the predetermined number of words back) are        fixed, and the corresponding digital ink is discarded and/or no        longer actively used for recognition purposes.

In word-mode, a user may perform a number of actions to make correctionsto recognized text. For instance, in one embodiment, as shown in FIG.5A, machines with pen digitizers that support hover will display abaseline 502 and a “delete-space” button 504 when the pen hovers over aspace (on touch machines without hover support, the user may need to gointo character-mode to delete spaces, as explained in further detailbelow). In an embodiment, the button 504 may be centered horizontallyover the space, and center-aligned vertically about the top edge of thewriting surface. Tapping the button 504 deletes the corresponding spaceand causes the words on each side of the space to be combined asillustrated in FIG. 5B. In the present example, after the user hastapped the button 504, the space between “ne” and “w” is deleted to formthe combined word “new.”

A user may also insert handwriting (which is converted to text) betweenwords of recognized text while in word-mode. To insert text betweenwords (or before the first word on the line), the user can simply startwriting in the space where the user wants to insert text. On pen down,the writing baseline is displayed, the space grows a predeterminedamount from the space size (e.g., an additional 30 pixels), and the newdigital ink is displayed on the writing surface. In another embodiment,if the user taps once in the space, instead of starting to write, thebaseline is displayed, the space grows a predetermined amount from thespace size, but no digital ink is displayed on the writing surface. Onthe next pen down, digital ink is displayed on the writing surface.

To make sure that the user can fit whatever they want to write betweenthe words, the baseline will continue to grow automatically each timedigital ink is added. This growing moves the words on the right-handside of the baseline further to the right. In an embodiment, when therightmost word won't fit on the line it may be moved to the next line.In some embodiments, scroll button(s) may be displayed on the right edgeof the writing surface to allow a user to scroll between multiple lines.In further embodiments, recognized text may be resized to allow the textto fit within a space available within the writing surface. Forinstance, the font of all text may be reduced to provide space for theuser's handwriting.

When a user is writing on the baseline between or before words ofrecognized text, the digital ink to recognized text conversion mayhappen similar to the conversion explained previously. Additionally,after the digital ink has been converted to recognized text, ashrink-timer (e.g., of 1000 ms) is started. The shrink-timer is stoppedand reset if new ink is added in the current space of the writingsurface. However, if the shrink-timer passes, the baseline starts toshrink to the default space size, and the text to the right of the spacefollows it. In an embodiment, the shrinking speed is slow to begin with(e.g., to not stress the user if he/she wants to write in the space butwaited a little too long), and accelerates until the size of the spaceapproaches the word space size. Toward the end, the shrinking will slowdown and come to a smooth stop as the space has been restored to thedefault word space size. FIGS. 6A and 6B provide graphs illustrating howthe shrinking progresses over time. The first graph 602 in FIG. 6A showsthat the space width is initially reduced slowly, then the space isreduced quicker as time passes, and finally the space is reduced slowlyagain as the space approaches the default space size. The second graph604 in FIG. 6B illustrates the shrink speed over time.

The addition of handwriting between words of recognized text isillustrated in FIG. 7A through FIG. 7C. Initially, as shown in FIG. 7A,a user begins writing the word “new” between the words “the” and“writing.” A base line 702 is displayed indicating to the user where theuse can insert ink. Additionally, the space between the wordsautomatically continues to grow as the user writes to ensure that theuser can fit the writing in the space. As the space grows, the words tothe right of the space (i.e., “writing” and “surface”) are moved to theright. As shown in FIG. 7B, the digital ink is recognized and convertedto recognition text. When the user has stopped writing and the shrinktimer has passed, the space begins to shrink back to the default wordspace size with the words to the right (i.e., “writing” and “surface”)following. As shown in FIG. 7C, when the space has shrunk back to thedefault word space size, the baseline in that space disappears.

In an embodiment, a heuristic may be used to prevent a space fromauto-growing when the user wants to perform a gesture (e.g., aleft-to-right scratch-out) that starts in a space. The space will notstart to auto-grow until the heuristic has determined that the user iswriting a word as opposed to performing a gesture. Once this has beendetermined, the new digital ink is treated as a word. In one embodiment,the heuristic analyzes the number of pen-packets over a stroke distanceto provide an indication of how fast the user is moving the pen. Thedetermination of whether the user is writing a word or performing agesture may then be determined based at least in part on the pen speed.

If the writing surface is being used with a digitizer that supportshover, when the pen is hovered over a word box, in one embodiment, avisual treatment may be provided to give an impression that the word boxis button-like and can be tapped. For instance, as shown in FIG. 8, whenthe pen is hovered over the word box for the word “new,” a word boxborder 802 is provided and the text color for the word “new” is changed.Additionally, a clear button 804 is displayed over the word box. If theuser selects the clear button 804, the corresponding word is removed. Inone embodiment, a baseline the size of the word box will appear in thespace remaining after the word is removed (e.g., in the event the userwould like to insert handwriting at that location). After a shrink-timerhas passed, the space will start to shrink as described above. If theuser clears the last word on a line, the word will be deleted and thebaseline will simply extend from the word box to the left of the deletedword to the end of the writing surface. The user can also clear one ormore words of recognized text by using a scratch-out gesture.Additionally, in some embodiments, performing a scratch-out gesture ondigital ink before it is converted will delete the digital ink.

In one embodiment, tapping on a word box will expand the word and showit in character-mode (as will be described in further detail below). Asnoted above, in some embodiments such as that shown in FIG. 8, a wordbox may be distinguished visually when a user hovers a pen over the wordbox, indicating to the user that the word box may be selected.Additionally or alternatively, the word boxes may have a visualtreatment without the use of hover, such that the word boxes aredistinguishable and to indicate to a user that they can be selected. Byway of example only and not limitation, in an embodiment as shown inFIG. 9, a drop-down arrow 902 may be provided under each word box toindicate that each word may be selected.

Character-Mode

Character-mode is used to make corrections to a recognized word inlinewith other recognized text. As previously noted, in one embodiment, toget into character-mode for a word, the user may tap on the word orotherwise select the word when it is displayed in word-mode. When a userselects a word to enter character-mode, the word is displayed in acharacter-mode or correction user interface that allows a user to editthe word on an individual character basis. Also, a number of additionalfeatures may be presented to facilitate editing the selected word. Whena word is selected, the word is “zoomed in” by enlarging the display ofthe word to facilitate corrections to the selected word.

By way of illustration, FIG. 10 provides a screen display showing awriting area in which the user has selected the word “wintry,” causingthe word to be presented in character-mode. As depicted in FIG. 10, thedisplay of the selected word is increased in size to allow thecharacters of the words to be displayed as individual boxes and tofacilitate accepting a user's handwriting.

In character-mode, the user can overwrite characters directly. Forinstance, as illustrated in FIG. 11, the user has overwritten the letter“i” with the letter “r.” The digital ink for the new character isrecognized, and the overwritten character is replaced with the newlyrecognized character. In an embodiment, the characters of the text incharacter-mode may be displayed differently from non-selected text(e.g., displaying the characters in light grey or a different color) toindicate that the user can overwrite the characters directly. A user mayalso perform a scratch-out gesture on one or more characters within thecharacter-mode user interface. A scratch-out gesture intersecting apredetermined portion (e.g., 80% or more) of a character box clears thebox. Multiple characters can also be cleared with one stroke if thestroke intersects at least a predetermined portion (e.g., 50% or more)of a character box.

Referring again to FIG. 10, a number of recognition alternates 1002 maybe provided for the selected word. For instance, in the present example,the alternates 1002 include, “Wintry,” “winery,” “writers,” and“writing.” If the user selects one of the recognition alternates, theword will be replaced with the selected alternate. In some embodiments,the character-mode user interface will be also be closed in response tothe selection (returning to word-mode). The character-mode userinterface may also include a clear button 1004. If the user selects theclear button 1004, the word will be deleted, and in some embodiments,the character-mode user interface will be closed.

Spaces 1006 may be presented on either side of the selected word incharacter-mode. In some embodiments, a delete space button 1008 may bepresented with each space 1006. If a user selects a delete space button1008, the corresponding space 1006 is deleted and the word incharacter-mode is combined with the word on the corresponding side ofthe selected button 1008. For instance, if the user were to select thedelete space button 1008 corresponding with the space 1006 to the right,the word “wintry” would be combined with the word “surface.” In someembodiments, the combined word is presented in character-mode, while inother embodiments the word is presented in word-mode. The user can alsoprepend or append characters in an open space 1006 in front of or behindthe word. After a character is written in an open space 1006, thecharacter-mode user interface will grow to provide a new open spacebefore or after the new character. For instance, in one embodiment, thecharacter-mode user interface may grow one space to the left if the userprepends a character, and grow one space to the right if the userappends a character after the word. This approach would avoid all thecharacters from moving. If there isn't enough space to expand in thedesired direction without adding scroll buttons (e.g., as described infurther detail below), then the character-mode user interface may expandin the other direction.

The character-mode user interface may also include a menu button 1010 toprovide further features. An illustrative menu 1202 is provided in FIG.12. In the present example, the menu includes an option 1204 to add theword shown in character-mode to or remove the word from a dictionary. Ifthe word already exists in the dictionary, an option to remove it willbe available. Selecting this option will remove the word from thedictionary and replace the menu item with an option to add the word tothe dictionary. If the word wasn't in the dictionary initially, thebehavior would be the opposite. The menu 1202 may also include an option1206 to access gesture help and other training associated with thewriting surface. In various embodiments of the invention, the menu 1202may further include a variety of additional features not shown.

In embodiments, the character-mode user interface also includes a numberof single-character features to facilitate a user's editing/correctionof the recognized word. In one embodiment, such as that shown in FIG.13, an expander button 1302 may be provided with one or more charactersto allow the user to access the further single-character features. On adevice that supports hover, the expander button 1302 for each charactermay typically be hidden, and the expander button 1302 for a character isdisplayed when a user hovers the pen over the corresponding characterbox. In some embodiments (e.g., on a device that doesn't support hover),an expander button 1302 for each character may be continuously displayedwith each character box while in character-mode.

When the user taps on the expander 1302 for one of the characters incharacter-mode, a variety of single-character features for that specificcharacter are provided such that as shown in the screen display of FIG.14. For instance, a number of alternate suggestions 1402 are providedfor the corresponding character. In the present example, the letters“W,” “v,” and “u” are provided as alternate suggestions 1402. If theuser selects an alternative, the character is replaced with the selectedalternate. If the user selects a delete button 1404, the currentcharacter is removed, and in some embodiments, all the characters to theright are moved one step to the left. Selecting the clear button 1406empties the content of the character box, leaving a space in which theuser may write a character. The space icons 1408 allow a space to beadded to the left or right of the selected character box. The space maybe added according to the same rules as for the growing when prependingor appending characters by using empty characters boxes described above.In some embodiments, the single-character features are no longerdisplayed in response to one of the above-noted user actions. Thesingle-character features for the current character may also be removedby the user selecting the upside-down expander icon 1410 for the currentcharacter, by the user switching directly to another character, or byanother action in the character-mode (for example overwriting anothercharacter). When the single-character features get closed, thecharacter-mode menu for the whole word will reappear.

Selecting any of the other word boxes will switch to showingcharacter-mode for the newly selected word instead. For instance, if theuser were to select the word “surface,” the word “wintry” would nolonger be shown in the character-mode user interface, and the word“surface” would now be shown in the character-mode user interface. Auser may exit character-mode and return to word-mode in a number ofdifferent ways within various embodiments of the invention. Forinstance, the user may exit character-mode by selecting an exitcharacter-mode button 1012 in FIG. 10. Additionally, the user may exitcharacter-mode by simply starting to write at the baseline, tapping onceat the baseline (without adding ink), or tapping at another location.Further, in some embodiments, character-mode may automatically be closedbased on some other user action such as selecting a word alternate.

In some embodiments of the present invention, the system may providereal-time suggested changes for a word based on a user's handwrittenediting while in character-mode. This allows a user to quickly correct amisrecognized word with very few character corrections. When the usermakes character corrections, the digital ink will be re-recognized withthe constraint(s) provided by the user's correction(s). When a new topalternate is available, it may be used to automatically replace the oldword. In some cases (e.g., when an active digitizer is employed), there-recognition and replacement may be triggered when the cursor movesout of the character box, the pen moves out screen range, or an overalltimer is passed. In other cases (e.g., when a touch digitizer isemployed), re-recognition and replacement may be performed after eachink stroke (on every pen up).

FIGS. 15A through 15D illustrate an automatic word replacement based ona handwritten correction in accordance with an embodiment of the presentinvention. Referring initially to FIG. 15A, a user is writing “This iswonderful.” The words “This” and “is” have already been converted totext, and the user is writing the word “wonderful.” As shown in FIG.15B, the recognizer misrecognizes the user's handwriting for the word“wonderful” as “wanderer.” Recognizing that the word has beenmisrecognized, the user may select the word “wanderer,” causing the wordto be displayed in character-mode as shown in FIG. 15C. As shown in FIG.15C, the user overwrites the character “a” with the character “o.” Therecognizer uses this information to constrain the recognition of theink. By doing so, the recognizer determines that the word “wonderful” isa top alternate. As shown in FIG. 15D, the rest of the word isautomatically updated to provide the best suggestion of what the usertried to write with the constraint of an “o” as the second character.

In an embodiment, the automatic word correction assumes that the userstarts a correction from the left to the right. Accordingly, everythingto the left of the rightmost corrected character is assumed to beaccepted as correct by the user, and will be fixed when an updatedrecognition suggestion is provided.

In some cases, the replacement word provided by the automatic wordcorrection may differ from what the user intended. Accordingly, in anembodiment, a revert entry option may be provided in the correction userinterface. If a user selects the revert entry option, the text isreverted to the text that would have been in the correction area if theautomatic word correction was not applied. For instance, with referenceto FIGS. 15C and 15D, after the automatic word correction has beenapplied, a revert entry option 1502 is provided with an indication ofthe word that would have resulted after the user's edit had theautomatic word correction not been applied. In the present example, theword “wonderer” is indicated with the revert entry option 1502,reflecting the user's change of the character “a” in “wanderer” to an“o” shown in FIG. 15C. If the user were to select the revert entryoption 1502, the word “wonderful” would be replaced with the word“wonderer” in the correction user interface.

Transitioning from Word-Mode to Character-Mode

In some embodiments, when transitioning from word-mode tocharacter-mode, the selected word is displayed in character-mode usingmore space than in word-mode to provide a fuller user interface for theword to facilitate receiving handwriting input and other selections fromthe user to make corrections. To create space within the writing surfacefor the character-mode user interface, in an embodiment, the font sizefor the other text is reduced. In some embodiments of the invention, thefont size of all other text may be reduced to a common size.Additionally, in some cases, the font may be made narrower as well assmaller to provide more space for the character-mode box. In someembodiments, the other words may be moved to provide space for the wordin the correction user interface. In further embodiments, the word space(i.e., the space between words) may additionally or alternatively bereduced to provide space for word in character-mode. In still furtherembodiments, scroll bars may be used to allow text to no longer bedisplayed within the writing surface but may be accessed by a user byemploying the scroll bars. Any and all such variations of providingspace within a writing surface for the word in character-mode may beemployed within the scope of embodiments of the present invention.

By way of example and not limitation, space may be provided for aselected word in character-mode as follows. Space is provided around thecharacter-mode box by reducing the font-size and word spacing for theother text. In an embodiment, the resizing will be consistent for allother text, including text on both sides of the character-mode box, ifapplicable. The new font size is determined by how small size the textto the left of the character-mode box has to be reduced to provide spacefor the character-mode box. The character-mode box is center-alignedaround the original location of the selected word if there is enoughspace on the left side. If reducing the surrounding text to a minimumfont size doesn't provide enough space, the character-mode box isleft-aligned as far as possible. The character-mode box is moved only sofar to the right so that the left side is aligned with where the leftedge of the corresponding word box was before it was selected. If thereis not enough room to fit the text to the left of the character-modebox, even after the font size is set to the minimum, then as much asneeded of this text is scrolled out of view to the left and a horizontalscroll button is provided. Text to the right of the character-mode boxis moved to the right. If there isn't enough room on to move the text tothe right, then the text may be moved out of view to the right and ahorizontal scroll button will appear. In some cases, the text may bemoved to the next line. When the user closes character-mode, the othertext may be moved back to its original location and original sizing. Insome embodiments, if enough room is not available to fit all the text onthe line, the text may flow to the next line. If there's no line below,a new one may be added. If there isn't room to display another lineunder the current one, then the line may be added and a vertical scrollbutton may be provided for scrolling vertically.

Referring to FIGS. 16A through 16G, examples of adjusting font size,word positions, word spacing, and use of scrolls are illustrated.Initially, FIG. 16A depicts four recognized words at their respectiveoriginal sizes and locations while in word-mode. As shown in FIG. 16A,when a user selects “Word 1,” the word is presented in character-mode.As seen in FIGS. 16A and 16B, the character-mode user interface for“Word 1” requires more space than that required by “Word 1” inword-mode. To provide space for this expansion, the other words, “Word2,” “Word 3,” and “Word 4” are moved to the right and the font of thesewords is reduced.

FIG. 16C illustrates a user selection of “Word 2,” causing the word tobe displayed in character-mode. As shown in FIG. 16C, the font size ofthe other words, “Word 1,” “Word 3,” and “Word 4” is reduced. There isnot enough room on the left side of “Word 2” for the character-mode boxto be completely center-aligned to the word's word-mode location.Accordingly, the character-mode box is left-aligned to “Word 1.”Additionally, “Word 3” and “Word 4” are moved to the right toaccommodate the space required for the character-mode box.

In FIG. 16D, the user has selected “Word 3,” causing the word to bedisplayed in character-mode. In the illustrated embodiment of FIG. 16D,when “Word 3” is presented in character-mode, there no longer issufficient space on the line for “Word 4” to be displayed even when thefont size of the other words have been reduced. Accordingly, in thepresent embodiment, “Word 4” is shown as being partially out of view anda right horizontal scroll button 1602 is provided. If the user selectsthe right scroll button 1602, the text is scrolled, causing “Word 4” tocome into view, “Word 1” to move partially out of view, and a lefthorizontal scroll button 1604 to be displayed, as shown in FIG. 16E.

As shown in FIG. 16F, a user may select the last word (i.e., “Word 4”)on the line. In one embodiment, if there is enough space available onthe line, the character-mode box is center-aligned on the position ofthe text in word-mode. Otherwise, the character-mode box is left-alignedon the preceding word (i.e., “Word 3”). In some embodiments, such asthat shown in FIG. 16G, there may not be enough room available when thelast word is displayed in character-mode. In this example, “Word 1” ismoved out of view and the left horizontal display button 1604 ispresented.

As noted previously, in some embodiments, words may be moved to a newline. In some cases, vertical scroll bars may be provided to scroll tolines that are not within the current view. For instance, in FIG. 16H,“Word 3” is shown in character-mode resulting in insufficient spacebeing available on the current line for all the words. Accordingly,“Word” 4 has been moved to the next line, which is out of view, and adown vertical scroll button 1606 has been provided.

Exemplary Methods

Turning to FIG. 17, a flow diagram is provided illustrating a method1700 for automatically converting digital ink to text in accordance withan embodiment of the present invention. Initially, as shown at block1702, handwriting input is received as a user writes using a handheldwriting device, such as an electronic pen or stylus. Based on thehandwriting input, digital ink is displayed, as shown at block 1704. Inaccordance with an embodiment of the present invention in which adigitizer is integrated with, overlays, or otherwise appended to adisplay device, the digital ink may appear on the display devicecorresponding with a location at which the user writes using thehandheld writing device.

The handwriting input is sent to and analyzed by a recognizer, as shownat block 1706. The recognizer analyzes the handwriting input to identifyone or more words of text corresponding with the handwriting input.Additionally, the systems monitors for the occurrence of one or morepredetermined events that indicate the display of the digital ink shouldbe replaced with the display of text from the recognizer, as shown atblock 1708. As noted above, in an embodiment, the system attempts toconvert the digital ink to text as the user finishes writing each wordand/or begins writing another word. In one embodiment, the systemmonitors for the occurrence of predetermined events associated withthree triggers: a distance-based trigger; a recognition-based trigger;and an overall timer-based trigger.

In accordance with the distance-based trigger, the digital ink isconverted to text when the handheld writing device is moved a particulardistance from the handwriting input. In an embodiment, thedistance-based trigger includes both a short-distance trigger and along-distance trigger. The short-distance trigger is used to triggerconversion when the handheld writing device is moved past a firstdistance for a predefined period of time. In an embodiment, the firstdistance and/or the predefined period of time associated with theshort-distance trigger may be adapted based on a user's handwriting, forinstance, by determining an average distance between words in the user'shandwriting and/or determining an average time between writing words.The long-distance trigger is user to trigger conversion when thehandheld writing device is moved past a second distance that is greaterthan the first distance associated with the short-distance trigger.

The recognition-based trigger is based on the recognition results fromthe recognizer. In particular, the recognition-based trigger may betime-based, in which if a word doesn't change in the recognition resultsfor a predetermined period of time, the conversion from digital ink totext is triggered. In accordance with the overall timer-based trigger,the digital ink is converted to text when no handwriting input isreceived from the handheld writing device for a predetermined overallperiod of time.

As shown at block 1710, it is determined whether a predetermined eventhas occurred for triggering the automatic conversion of digital ink totext. If a predetermined event has not occurred, the system continues tomonitor for the occurrence of a predetermined event as represented bythe return to block 1708. Otherwise, if a predetermined event hasoccurred, the display of the digital ink is replaced with recognizedtext for the digital ink, as shown at block 1712. The text is displayedat a location corresponding with the location of the digital ink andinline with other text and/or other digital ink from further handwritinginput as the user continues to write.

Referring now to FIG. 18, a flow diagram is provided illustrating amethod 1800 for presenting a word of text in a correction user interfacein accordance with an embodiment of the present invention. Initially, asshown at block 1802, at least two words of text are displayed that havebeen converted from digital ink corresponding with handwriting input.For instance, the text may have been converted from digital ink in amanner similar to that described above for the method 1700 withreference to FIG. 17.

A selection of a word of the text is received, as shown at block 1804.The word may be selected in a variety of different manners withinvarious embodiments of the present invention. By way of example only andnot limitation, the word be selected by a user tapping a handheldwriting device, such as an electronic pen or stylus, on the word. Insome cases, the word may be selected by tapping an icon, button, orother user interface widget associated with the word. Any and all suchvariations are contemplated to be within the scope of embodiments of thepresent invention.

In response to the selection, the word is displayed in a correction userinterface, as shown at block 1806. The correction user interfaceprovides an enlarged display of the word that replaces the originaldisplay of the word. Accordingly, the enlarged displaying of the word isinline with other words within the text. The correction user interfaceallows the user to edit the word on an individual character basis aswell as provides other features and options for correcting the word. Forinstance, the correction user interface may provide alternate words forthe selected word.

As shown at block 1808, the display of the other words within the textis adjusted to provide space for the enlarged display of the selectedword in the correction user interface. In various embodiments of thepresent invention, the other words may be resized and/or moved toprovide space for the correction user interface. In some embodiments,the spacing between the words may be reduced. In further embodiments,scroll bars may be provided if one or more words are moved outside ofthe display area. By employing the scroll bars, the user may bring thosewords back into view.

As can be understood, embodiments of the present invention provide forinline handwriting recognition and correction. The present invention hasbeen described in relation to particular embodiments, which are intendedin all respects to be illustrative rather than restrictive. Alternativeembodiments will become apparent to those of ordinary skill in the artto which the present invention pertains without departing from itsscope.

From the foregoing, it will be seen that this invention is one welladapted to attain all the ends and objects set forth above, togetherwith other advantages which are obvious and inherent to the system andmethod. It will be understood that certain features and subcombinationsare of utility and may be employed without reference to other featuresand subcombinations. This is contemplated by and is within the scope ofthe claims.

What is claimed is:
 1. One or more computer-storage media devicesembodying computer-useable instructions for performing a methodcomprising: receiving input corresponding with user handwriting from ahandheld writing device; displaying digital ink representing the userhandwriting based on the input; analyzing the input using a recognizerto identify one or more words as recognition text for the digital ink;employing at least three independent triggers to initiate conversion ofdisplay of the digital ink to display of the recognition text, whereinthe at least three independent triggers include a distance-basedtrigger, a recognition-based trigger, and an overall timer-basedtrigger; determining that at least one of the at least three independenttriggers has been satisfied, wherein satisfaction of any of the at leastthree independent triggers initiates conversion of display of thedigital ink to display of the recognition text independent ofsatisfaction of the other independent triggers; and displaying therecognition text in place of the digital ink.
 2. The one or morecomputer-storage media devices of claim 1, wherein the distance-basedtrigger includes a short-distance trigger and a longer-distance trigger,wherein the short-distance trigger is satisfied when the handheldwriting device is moved past a first distance for more than a predefinedperiod of time, and wherein the long-distance trigger is satisfied whenthe handheld writing device is moved past a second distance, the seconddistance being greater than the first distance.
 3. The one or morecomputer-storage media devices of claim 2, wherein the first distance isbased on an average spacing between words identified from previous userhandwriting.
 4. The one or more computer-storage media devices of claim1, wherein displaying the recognition text in place of the digital inkcomprises gradually fading out the digital ink as the recognition textis displayed.
 5. The one or more computer-storage media devices of claim1, wherein the method further comprises saving the digital ink after therecognition text has been displayed in place of the digital ink andusing the digital ink with new digital ink for recognition purposes. 6.The one or more computer-storage media devices of claim 5, wherein thedigital ink is no longer used for recognition purposes when recognitionresults for the digital ink have not changed for a predetermined periodof time.
 7. The one or more computer-storage media devices of claim 1,wherein the method further comprises: receiving additional inputcorresponding with additional user handwriting from the handheld writingdevice, wherein the additional input is received between two words ofrecognition text; expanding a space between the two words to providespace for receiving the additional user handwriting; displayingadditional digital ink based on the additional input in the spacebetween the two words; analyzing the additional input using therecognizer to identify one or more additional words as additionalrecognition text for the additional digital ink; employing the at leastthree triggers to determine when to convert display of the additionaldigital ink to display of the additional recognition text; determiningthat at least one of the at least three triggers has been satisfiedindicating to convert display of the additional digital ink to theadditional recognition text; and displaying the additional recognitiontext in place of the additional digital ink.
 8. One or morecomputer-storage media devices embodying computer-useable instructionsfor performing a method comprising: displaying text including two ormore words, the text having been previously converted from digital inkcorresponding with handwriting input; receiving a user selection of aword within the text; responsive to the user selection, displaying theword in a correction user interface; displaying, within the correctionuser interface, an enlarged display of the word in place of an originaldisplay of the word in a manner in which the enlarged display of theword is inline with other words within the text, wherein acharacter-mode interface is displayed as part of the correction userinterface allowing correction of the word on an individual characterbasis, wherein individual characters are separated within the correctionuser interface by visible markings; and adjusting the display of theother words within the text to provide space for displaying thecorrection user interface including the enlarged display of the word,wherein the correction user interface does not occlude the other wordswithin the text.
 9. The one or more computer-storage media devices ofclaim 8, wherein adjusting the display of the other words within thetext comprises at least one of resizing at least one of the other wordsand moving at least one of the other words.
 10. The one or morecomputer-storage media devices of claim 9, wherein adjusting the displayof the other words further comprises providing at least one of ahorizontal scroll bar and a vertical scroll bar.
 11. The one or morecomputer-storage media devices of claim 8, wherein adjusting the displayof the other words comprises reducing spacing between the other words.12. The one or more computer-storage media devices of claim 8, whereinthe method further comprises: receiving handwriting input at a locationcorresponding with a character of the word within the correction userinterface; displaying digital ink corresponding with the handwritinginput; analyzing the handwriting input using a recognizer to identify anew character corresponding with the digital ink; and replacing displayof the character with the new character.
 13. The one or morecomputer-storage media devices of claim 8, wherein displaying thecorrection user interface includes displaying one or more alternatewords for the word, and wherein the method further comprises: receivinga user selection of one of the one or more alternate words; andreplacing the word with the selected alternate word.
 14. The one or morecomputer-storage media devices of claim 8, wherein the method furthercomprises: receiving a user selection of a character within the word;and presenting one or more single-character features for editing theselected character.
 15. The one or more computer-storage media devicesof claim 8, wherein the method further comprises: receiving handwritinginput at a location corresponding with a character of the word withinthe correction user interface; displaying digital ink corresponding withthe handwriting input; analyzing at least the handwriting input using arecognizer to identify an alternate word for the word; and automaticallyreplacing display of the word with display of the alternate word. 16.One or more computer-storage media devices embodying computer-useableinstructions for performing a method comprising: receiving handwritinginput corresponding with a handheld writing device being moved relativeto a display surface; displaying digital ink at a location on a displaycorresponding with a location of the display surface at which thehandwriting input is received; analyzing the handwriting input using arecognizer to identify one or more words of text corresponding with thehandwriting input; replacing display of the digital ink with the text asthe user continues to provide additional handwriting input using thehandheld writing device, wherein the text is displayed inline withadditional digital ink corresponding with the additional handwritinginput; receiving input to present a word of the text in a charactercorrection interface, the character correction interface providing anenlarged display of the word in place of an original display of the wordto facilitate receiving further handwriting input to correct the word,wherein the character correction interface allows correction of the wordon an individual character basis, wherein individual characters areseparated within the correction user interface by visible markings;displaying the word in the character correction interface by replacingthe original display of the word with the enlarged display of the word;and adjusting display of other words within the text to provide spacefor the enlarged display of the word, wherein the correction userinterface does not occlude the other words within the text.
 17. The oneor more computer-storage media devices of claim 16, wherein replacingthe display of the digital ink with the text comprises: monitoring foran occurrence of at least one predetermined event; and upon detectingthe occurrence of a predetermined event, displaying the text in place ofthe digital ink.
 18. The one or more computer-storage media devices ofclaim 17, wherein monitoring for the occurrence of at least onepredetermined event comprises: monitoring for movement of the handheldwriting device a predetermined distance from the location of the digitalink; monitoring for stabilization of recognition results for thehandwriting input for a predetermined period of time; and monitoring forfurther handwriting input within a second predetermined period of time.19. The one or more computer-storage media devices of claim 16, whereinthe method further comprises: receiving further handwriting input at alocation corresponding with a character of the word within the charactercorrection user interface; displaying further digital ink correspondingwith the further handwriting input; analyzing at least the furtherhandwriting input using a recognizer to identify an alternate word forthe word; and automatically replacing display of the word with displayof the alternate word.